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Abstract. The ordering constraint along epipolar lines is a powerful constraint that has 
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j*«^ ordering constraint, not restricted to epipolar lines. We prove several properties of the 

generalized ordering constraint and of the "forbidden zone", the set of matches that would 
violate the constraint. We consider both the orthographic and the perspective projection 
case, the latter for a simplified but standard stereo geometry. The disparity gradient limit 
found in the human stereo system may be related to a form of the ordering constraint. To 
illustrate our analysis we outline a simple algorithm that exploits the generalized ordering 
constraint for matching contours of wireframe objects. We also show that the use of the 
generalized ordering constraint implies several other stereo matching constraints: a) the 
ordering constraint along epipolar lines, b) figural continuity, c) Binford's cross-product 
constraint, d) Mayhew and Frisby's figural continuity constraint. We finally discuss ways of 
extending the algorithm to arbitrary 3-D objects. 



/■■% 



This report describes research dene within the Artificial Intelligence Laboratory and the 
Center for Biological Information Processing (Whitaker College) at the Massachusetts 
Institute of Technology. Support for the A. I. Laboratory's research in artificial intelligence 
is provided in part by the Advanced Research Projects Agency of the Department of 
Defense under Office of Naval Research contract N00014-80-C-C505. The Center's support 
is provided in part by the Sloan Foundation and in part by Whitaker College. 



© Massachusetts Institute of Technology, 1084 



/•*•% 



Screen I 



Screen 2 




Figure 1 The geometry of the epipolar lines. The plane defined by the two foci, f\ and 
F-i, and a point P cuts the two image planes in the corresponding epipolar lines li and l 2 . 
As we vary the position of P we produce a 1-parameter family of epipolar lines on each 
screen. The lines are not parallel to one another, and it can be seen that the epipolar lines 
on screen 1 all intersect at the point where the line l ? il<\ hits screen 1. 



1. Introduction 
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The problem of stereo matching is ill-posed and underdetermined: constraints are needed 
(a) to make the solution unique and (b) to reduce the search problem among possible 
matches. 

Marr and Poggio (1979) originally identified two important constraints: (1) uniqueness, that 
is, an element in one image in general only corresponds with a single element in the other 
image, and (2) continuity, that is, stereo disparity varies smoothly almost everywhere in the 
image. These constraints are powerful because they do not depend on the specific properties 
of the scene but on general properties of the stereo geometry. Marr and Poggio (1979) 
proposed a stereo matching algorithm, further developed by Grimson (1981, 1984), which 
incorporates the uniqueness and continuity constraints to match zero-crossing descriptions 
computed at different scales. An ordering constraint along epipolar lines has been exploited, 
both implicitly and explicitly, in several computer algorithms for stereo matching, as a special 
instance of the continuity constraint. Epipolar lines in the two images are lines on which 
corresponding points lie. The projections of a point P in space lie on the plane defined 
by P and the two camera foci and, as a consequence, on the two lines defined by the 
intersection of this plane with the two image planes (see figure 1). 

This implies that the matching problem can be reduced to a one-dimensional search if the 
epipolars are known. Most algorithms assume that the epipolar geometry is known (from a 
known camera geometry) and that the images are registered. Furthermore, the ordering of 
edges or other features is usually preserved by stereo projection along epipolar lines (that 
is, if feature A is to the left of feature B in the left stereo image, then this spatial relationship 
is maintained in the right stereo image). The ordering constraint along epipolar lines follows 
from the continuity of surfaces and the assumption of opacity. As originally suggested by 
Baker (1982) the ordering constraint is violated in situations such as figure 2. Recently, 
Verri (1984) has discussed the role of the "forbidden zone", where the ordering constraint 
is violated (Krol and van de Grind, 1982 first introduced the notion of forbidden zone). The 
forbidden zone associated with each point of the visible surface is a set of points in space 
that would have images violating the ordering constraint. If any point in the forbidden zone 
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Figure 2 The forbidden zone associated with point A is represented by the dashed region. 
Any point in this region such as B has projections which violate the ordering constraint 
relative to A 

would be connected to the first point by an opaque surface the two images would "see" 
opposite sides of the surface. 

/"""N This ordering constraint can be exploited to reduce the complexity of the search for matching 

features, and to eliminate false matches. Interestingly, there is preliminary evidence that the 
ordering contraint (and perhaps a stronger form of it) may be implemented in the human 
stereo system (Burt & Julesz 1980). The human system, However, must often cope with 
situations in which the images are not precisely registered. Furthermore, physical edges 
are inherently two-dimensional (a property that is not exploited by the epipolar ordering 
constraint). It is therefore natural to ask whether the ordering constraint can be generalized 
from epipolar lines 1 . More precisely, can an ordering constraint be formulated that is 
independent of the epipolar geometry? The simplest use of such a constraint would be 
when either the epipolar lines are unknown or their estimation is affected by errors. 

In this paper we show that it is indeed possible to generalize the ordering constraint. We 
also give an analytic definition of the forbidden zone and characterize its properties. We 
show then that the generalized ordering constraint implies several other constraints that 
have been exploited in stereo matching. An algorithm based on this constraint has not been 
implemented yet, but we discuss its advantages and its limitations. 

The plan of the paper is as follows. Throughout the paper, unless specifically stated, we 
consider a stereo geometry in which the two image planes have the same vertical unit 
vectors. . This simple geometry is fully representative for most practical applications and it 
represents a good approximation for human binocular geometry (Longuet-Higgins, 1982). In 
section 2 we assume orthographic projection. We prove a simple relationship between the 
two images of a 3D curve that leads to a generalization of the standard ordering constraint. 
This relationship allows us to identify special points in the images that correspond uniquely 
to the same physical point in the object curve. The Generalized Ordering Constraint (GOC) 
implies several of the specific constraints listed by Baker et al. (1983) , Mayhew and Frisby 
(1981) (see their figural continuity constraint), and Ohta and Kanade (1983). The ordering 
,/****• constraint breaks down when the object curve enters the forbidden zone. We define the 

1 This question was first brought to our attention by Dr. V. Torre 
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that slightly weaker results can be derived for perspective projection (this includes the 
ordering constraint along epipolar lines as a very special case). These results again break 
down for the forbidden zone, which is more complex than in the orthographic case. In 
section 3, we characterize fully the forbidden zone for the perspective case and prove several 
results about it. The boundaries of the forbidden zone correspond to the so called Panum 
limiting case in the psychophysics of stereo, when one line of sight just grazes the surface. 
Burt and Julesz' results suggest that human stereovision is limited to a smaller disparity 
gradient. In. section 4 we derive the equations for the physical surfaces corresponding to 
this disparity gradient limit. In section 5 we show how the generalized ordering constraint 
implies other stereo matching constraints. In section 6 we outline an algorithm for stereo 
based on matching contours. From a single contour the algorithm retrieves the viewing 
parameters and unambiguously matches points along the contour using the generalized 
ordering constraint. The constraint of figural continuity (Mayhew and Frisby, 1981) follows 
from the generalized ordering constraint and is implicitly implemented in our algorithm. In 
Appendix 2 we derive an explicit solution of the equations for the viewing parameters in the 
case of the stereo geometry suggested by Longuet-Higgins, using only two points. 



2. The Ordering Constraint and the Forbidden Zone for Orthographic 
Projection 



We assume a stereo-imaging geometry of the type proposed by Lonyuet-Higgins (1982) in 
iT\ which the planes of the horizontal meridians of the two eyes coincide. Hence, the relative 

orientation of the two eyes is defined by one parameter only. We also assume orthographic 
projection. We show that for any part of the object which does not lie in the forbidden zone, 
there is a simple relationship between the images of the object in the two eyes. This enables 
us to generalize the ordering constraint (Baker & Binford, 1981) and to identify features in 
the images that correspond to the same feature in the object. 

We define an orthonormal triad of vectors in each eye. By our restriction on the geometry, 
the vertical direction, k, is the same for each eye. The right eye has vectors i, j and k, 
where j is normal to the right image plane. Similarly the left eye has vectors i', j' and k, 
where j' is normal to the left image plane The convergence angle <d satisfies j • j' = costf 
(see Figure 3). We define coordinates X, Y and Z along the i, j and k axes respectively. 
Similarly, we let X' and Y' be coordinates along the i' and j' directions. Note that the 
origins of these systems of coordinates lies at the intersection of the two image planes and 
not at the focal points of the eyes. The following equations connect the triads of the two 
eyes: 

j • j ' = COS 1? 

i • i ' = cos i? 

i j' — -t-sini? 

We consider now a curve in the right image plane, parameterized by s (which is not the arc 
length) and written as 
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r R {s) = X(a)i + Z{ s )k (2.2) 

Under the assumption of orthographic projection, equation (2.2) is the image of an object 
curve given by r(.») = X(s)\ + Y(s)] + j?(a)k, 
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Figure 3 See text. 
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where 



r L ( 3 )=X'( a )\' + Z'{s)k 



X '(s) = r(a) • i ' = X(a) cos i? - Y(a) sin t? 
Z'(a) = r(a)-k = 2'(a) 



(2.3) 



(2.4) 



We partition the right image curve into intervals for which Z is a single valued function of 
X. These intervals are separated by points at which f£ = 0. For each such interval X can 
be chosen as the parameter and we can write the projection (in the right image plane) as 



r R (X) = X\ + Z{X)k 
Identification of s in equation (2.3) with X yields (for the left image) 

• r = X'{X)\' + Z'(X)k 
with 

X '(X) = X cos t? - Y{X) sin i? 
Z'{X) = Z(X) 

Let us now compute ^ from equation (2.4): 



(2.5) 



(2.6) 



(2.7) 



dX' . dY(X) . „ 
= cos v .„„ sin 17 



dX 



Note that (see Figure 4) 



dX 



dY cosi? 
dX < sinr? 



(2.8) 



(2.9) 



is the condition that curve never enters the "forbidden zone". Thus, if the curve never 
enters the forbidden zone, i.e., it satisfies the condition (2.9), then 
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Figure 4 Line (a) has gradient smaller than cold and therefore is not in the forbidden 
zone. Line (b) has gradient larger than cotti and lies in the forbidden zone. The two views 
are of "opposite" sides. 



dX' 
dX 



>o. 



(2.10) 



Let us consider now the slope of the curve in the two images for each partition. The chain 
rule yields 



dZ' _ dZ dX 
dX'~~ dX dX> 



(2.11) 



Thus, if the curve is outside the forbidden zone, (a) $£ and iff nave tne same sign (because 
of equation (2.10). Moreover, (b) the zeros of ||4 correspond to zeros of $£. Furthermore, 
(c) equation (2.10) gives an ordering constraint on matching corresponding points in the 
two curves, since within each partition ordered points on the right image curve correspond 
to points on the right image curve with the same order (because of monotonicity implied by 
equation (2.11)). 



The Forbidden Zone for Orthographic Projection 
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We now define the forbidden zone and show that it occurs precisely where the ordering 
constraints break down. 
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Figure 5 Points Ai and A 2 are the images of the physical point A ana have coordinates 
Xi and X 2 , respectively. 



/*"\ 



O 



Suppose the two screens, screen 1 and screen 2, are at an angle to each other as 
illustrated in Figures 5 and 6. Consider two points A and A' with coordinates (X"i,Yi) and 
(X[,Y[) respectively relative to screen 1. They will be projected to points A x and A[ on 
screen 1 with OAi — X x and OA[ = X[. They will be projected to points A 2 and A 2 on 
screen 2 with OA 2 = X 2 = Xi cos + Yi sin and OA' 2 = X£ = X[ co&6 + Y^smO. The 
ordering constraint will hold, provided 



X[ - Xi > 0, if and only ifx£ - X 2 > 



We have 



X' 2 -X 2 = [X[ -Xxjcos^l + -j 



y,' - y, 



X{-X x 
So a necessary and sufficient condition for (2.12) to hold is that 



tan<?} 



1 + 



Y[-Yi 
X[-X 1 



tan^ > 



(2.12) 



(2.13) 



(2.H) 



Note that we have -n/2 < < n/2 and hence cos 6 is always positive. 

Equation (2.14) can be interpreted as a condition on the gradient m of the straight line 

joining A to A'. Substituting m for £i^r- in (2.14) gives 



Xl-Xi 



m > — cot 



(2.15) 



which is the same as equation (2.9). 

It is easy to see from Figure 6 that if this condition is violated, screen 1 and screen 2 will 
see opposite sides of the line joining A to A'. This motivates the following definition of the 
forbidden zone: 

•A point A ' is in the forbidden zone of a point A if, and only if, the two screens see 
opposite sides of the straight line joining A' to A 

As we have shown above, the forbidden zone is precisely the region where the ordering 
constraint breaks down. 
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Figure 6 See text. 



Observe from Figure 6 that the definition of the forbidden zone is symmetric, that is, if A' 
is in the forbidden zone of A, then A' is in the forbidden zone of A. Note also that the 
forbidden zone of A depends only on the position of A and the angle between the two 
screens. This will not be the case when we consider perspective projection. In Section 3, 
we prove some results about the forbidden zone for perspective projection, which will also 
apply to orthographic projection. 



3. Perspective Projection 

We now consider the ordering constraints for perspective projection. 

The geometry is summarized in Figure 7. The two screens have unit normals a t and a 2 
and focal length m. The foci have position vectors fi, and f 2 relative to the origin 0. An 
arbitrary point A in space has position vector X relative to 0. 

The centers of screen 1 and screen 2 are r? and r§, where 



r° — f 



rij = f 2 — rna s 



The equations of the screens are 



r-a, 



t ■ &2 



U • «i 

U -a 2 



m, 
m, 



for screen 1 
for screen 2. 



Now consider the projection of a point X on screen 1. The line of projection is 



r(X) = X + X(fj - X) 



(3.1) 



(3.2) 



(3.3) 



From (3.2), this hits screen 1 at 
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X • o, + X(f i - X) • si = fi • BL t 



(3.4) 



We solve (3.4) to obtain 
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Figure 7 See text. 
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X-l = 



— ra 



and so X is projected to a point r x with 



fi 



(fi-X)- fil 



m(f j - X) 
(fi-X). ai 



(3.5) 



(3.6) 



To find the position X x of the image of X relative to the center of the screen, rather than 
relative to 0, we must subtract r? from r lt Hence, 



Xi = f! - r° 



o 



(fi-X). ftl 
Similarly, the projection on screen 2 is given by 



(((fi-X).a 1 )a 1 -(f 1 -X)> 



x,= 



m 



(fa-X).fl. 



•(((f 2 -X).a 2 )a 2 -(f 2 -X)> 



(3.7) 



(3.8) 



Now we restrict ourselves to the geometry used by Longuet-Higgins (1982). In this case, 
the vectors ai,a 2 ,f, and f 2 are coplanar and perpendicular to a vector k that we take to 
be in the z-direction. We can choose the origin and the z-axis so that 



fi = -f 2 = (/,0,0) 
We define angles 4>x and <£ 2 , such that 

Qj = (co8(£i,sin<£i,0) 
a 2 = (cos ^ 2 , sin ^2,0), 



(3.9) 



(3.10) 



and two vectors <x\ and «; orthogonal to «, and a? 
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Figure 8 See text. 
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aj = (sin <f>i, — cos0i,O) 
«2 = ( sir > 02 > — cos 02 » 0) 



(3.11) 



The vectors aj, «! and k form a right-handed triad for screen 1, as do a* 2 , a 2 and k for 
screen 2. 

Let the components of X be (X, Y, Z). Its projection Xi on screen 1 is given by (3.7), and 
can be written 



X 1 =(X 1 -«l)«l+(Xi-k)k 



(3.12) 



Note Xi ■ a x — 0, since Xi lies on the screen. Thus we can take X x = (Xj -«J) and 
Z x = (Xi • k) to be the cartesian coordinates of the image on screen 1. Substituting from 
(3.11), (3.9) and (3.7) gives 



*i = 



Zi = 



/ cos <j>i — X cos <f>Y —Y sin 0i 

mZ 
f cos <j>\ — X cos 4>i — y sin 0i 



(—/sin 0i + Xsin 0i — Y costf>i) 



Similarly, we have 



(3.13) 



fS. 



X, = 



— m 



^2 



/ cos 02 + X cos 02 + y sin 02 

—mZ 
f cos fa + X cos <fo + Y sin 02 



(/sin fa + Xsin 02 — y cos 02 ) 



(3.14) 



It is well-known that there is an ordering constraint along epipolar lines, and with our 
geometry, one such line occurs when Z — 0. This is illustrated by Figure 8, where X and 
X' lie in the Z ~ plane. The forbidden zone of X is the shaded region. Then, provided 
that X' lies outside the forbidden zone of X the images of X' on the two screens will be 
either both to the right of the image of X, as in Figure 8, or both to the left. 



/•""N 



f~\ 



POGGIO YUILLE 

tz 

Screen 1 



Screen 2 




Figure 9 See text. 

Now observe that both Xi and X 2 in (3.13) and (3.14) are independent of Z. Thus, if we 
remove the restriction Z — 0, we obtain a similar ordering constraint independent of whether 
X arid X' //e on the same epipolar line. Intuitively, if X is to the "right" of X' in space, then 
the projections of X will be to the "right" of the projections of X' on both screens. 

These results hold if, and only if, X' is not in the forbidden zone of X. The forbidden zone 
is complicated for perspective projections, and we discuss it in the next subsection. 

One obvious application of our generalized ordering constraint concerns two contours on 
f\ the two screens. It implies that there is an ordered mapping between the two contours and 

that the points at the end of the contours, where f£ = 0, must be identified. (See Figure 
9.) The ordering constraint along epipolar lines arises as a special case when we take the 
contours to be the epipolar lines. 

The Forbidden Zone for Perspective Projection. 

Since the X-component on the screen is independent of the ^-component in space, (3.13) 
and (3.14), the shape of the forbidden zone will be independent of Z. 

From Figure 10, it is clear that B' is in the forbidden zone of A if and only if the two screens 
see opposite sides of the straight line joining A to B'. 

The structure of the forbidden zone is more complicated for perspective than for orthographic 
projection. For orthographic projection, the angle subtended by A to the two screens was 
a fixed angle 0, independent of the position of A. For perspective, this is no longer the 
case. In this section, we characterize the forbidden zone algebraically and prove various 
desirable properties about it. 

Let the permissive zone of A be the complement of the forbidden zone of A. If B is in the 
permissive zone of A to the left of A we will write B left *A. Similarly, B right *A means B 
is in the permissive zone of A to the right of A. 

It is clear from Figure 10 that B left *A if and only if the angles t/>i and i/>2 are both positive. 
This will be true if and only if the ^-component of the cross-product of the vectors h\A 
with F-Ji and l<\A with l<\ B are both positive (since the lines are coplanar, the sign of the 
/^-component is the sign of sim/>o and sim/>i, respectively). 

The vectors F 2 A and h\ B are written 
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Figure 10 B' lies in the forbidden zone of A but B does not. 
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F 2 A^(X 1 + f,Y 1 ,0) 
F 2 B = (X 2 + f,Y 2 ,0) 



and their cross-product is 



FiA X Ft]} = {(X, + f)Y 2 - (X 2 + /)Y,}k 
Thus the point B is to the left of the line F 2 A if 

(X, + /)Y 2 - (X, + /)n > 
and to the right if 

{X t + f)Y 2 - (X 2 + /)n < 0. 
Similarly, B is to the left of F % A if 



(Xx - f)Y 2 - (X 2 - f)Y x > 



and to the right if 



(X, - f)Y* - (X 2 - /)n < 0. 
So we can have B left *A if and only if both 



(X t + f)Y 2 - (X 2 + /)Y X > 
(X, - f)Y 2 - (X 2 - /)K, > 



and B right* A if and only if both 



(X, + J)Y 2 - (X 2 + /)Y < 
(X, - f)Y 2 - (X 2 - f)Y x < 0' 



(3.15) 



(3.16). 



(3.17) 



(3.18) 



(3.19) 



(3.20) 



(3.21) 



(3.22) 
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Figure 1 1 Tfte line AB projects to a line on screen 1 and a point on screen 2. 

Observe that if Y 2 > Yi then if B is to the left of F X A it is automatically to the left of F 2 A. 

If Y 2 <Y U then if B is to the left of F 2 A it is automatically to the left of F X A. This is an 

alternative way of characterizing left* and right*. 

We now show that the forbidden zone has the desirable properties of symmetry and 

transitivity. 

For svmmetrv. we must show that B left* A is equivalent to A right* B. In other words, if B 

is in the permissive zone of A to the left, then A is in the permissive zone of B to the right 

and conversely. 

The conditions for B left* A axe given by (3.21). The condition for A right*B are, from (3.22) 

(X 2 + f)Y 1 -(X l +f)Y 2 <0 
(X 2 - /)y, -{X l -f)Y i <0 

The equation (3.21) and (3.23) are clearly equivalent and hence the result is proved. 

Now we consider transitivity. We want to show that if C left*C and B left* A then C left* A 

C left*/? means that F 2 C is to the left of F 2 B and F V C is to the left of FJi. From B left*,4 
we have F 2 B to the left of F 2 A and I'\B is to the left of F X A. So, F 2 C is to the left of F 2 B 
which is to the left of F 2 A and hence, F 2 C is to the left of F 2 A. Similarly, F X C is to the left 
of F X A. Thus C left*yt and the result holds. 



4. Limits to Fusion 



We have shown in the preceding sections that when a point B is inside the forbidden zone 
of a point A, different sides of the straight line joining A to B are projected to different 
screens. If B lies on the boundary of the forbidden zone of A then, as in Figure 11, one 
side of the line will be visible to one screen but the other screen will only see a point. This 
corresponds to Panum's Limiting case. 

f^ If the point B is moved to B' in the permissive region of A then the line AB' can be seen by 

both screen 1 and screen 2 although the projection to screen 2 will be very foreshortened. 
It should in principle be possible to fuse features on the line AB' despite this foreshortening. 
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Figure 12 See text 



f^\ 



In a recent paper, however, Burt and Julesz (1980) claim that there is a limit to the relative 
disparities in two screens that the human visual system can fuse. If this is so, then the 
boundaries of the forbidden zone may be a competence limit while the Burt-Julesz limit may 
be a performance limit. We have so far been assuming that the greatest slopes in space 
that we can see are bounded above by the forbidden zone and Burt-Julesz' results could 
tell us that this limit cannot always be attained. To investigate this, we ask what surfaces in 
space correspond to a limit of disparity gradient. We will see that they are straight lines. 

First, assume the system is fixating at infinity so the two screens are parallel. We illustrate 
this case by figure (12). Now let A be a point in the center of the visual field with coordinate 
(0, a). It projects to points A t and A 2 on the screens. These have components (/(l + *~), -m) 
and (-/(l + ^), -m) respectively. Let the relative disparity limit be denoted by a (Burt-Julesz 
would set a = 3). The relative disparity of two points in space is defined as the ratio of 
the differences of their images in the two screens (see Fig. 12). So the fusional limit would 
occur for points Q and P on the two screens, where Q is a distance X to the left of Ai and 
P is qX to the left of A 2 . So we have 



m. 



/■"S 



qQ = (f(l + -)-\,-m) 



TO, 



G£ = (-/(l + -)-«X,-m) 
a 



(4.1) 



The line QF X is given by 



where fi = rnf/a, and the line FF 2 is 



These lines intersect at a point where 



aX + n 



(* + /)• 



(4.2) 



(4.3) 
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Figure 13 The screens fixate at A with mergence angle 2$ 



A(i + <*) 

2(x + X(a - 1) 
2m/ 



(4.4) 



2/x + X(a - 1) 



Letting X vary, we obtain a curve passing through A which is the greatest clcpe ecrreopending 
to the Burt-Julesz limit. Eliminating X from (4.4) gives the curve 



-{^\)f( a -y) ( 4 - 5 ) 



which is a straight line. Observe that the gradient of the curve depends only on a and /. 
Setting a = 3 gives a gradient of ^f. 

We also perform the calculation when the visual system is fixating on A as in Figure 13. 

It is straightforward to modify the previous calculation by setting 

m' = m cos ^ (46) 

X ' = X cos <f> 

Note that in this case we have 

tan^=^. (4.7) 

a 

Since neither m or X appear in the final expression (4.5), the result is unaltered by the 
change in fixation point. As before, the gradient is proportional to f and by (4.7) it is hence 
proportional to cos^. 

5. The generalized ordering constraint and other constraints 

Arnold and Binford (1980) and Mayhew and Frisby (1981) suggested a figural continuity 

/*"> constraint for stereo matching: disparity is usually continuous along contours. This figural 

continuity constraint can be powerful in practical implementations as demonstrated by 
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f^. Grimson (1984) and Ohta and Kanade (1984). Our geometric model of matching implicitely 

implements figural continuity, because matching is performed along continuous contours 
under the constraint that the disparity gradient is not too large (i.e., the contour does not 
enter the forbidden zone). 

Binford (see Baker et al., 1983) suggested surface occlusion rules for making explicit opacity 
and non-opacity of surfaces. In particular the cross-product rule determines whether an 
hypothetized match between two corners lies in the forbidden zone. Again a rule such as 
this follows immediately from the generalized ordering constraint. 

6. Epipolar lines as a constraint for matching 

We now proceed to show that if the stereo geometry is known, perhaps by registering 
the screens from the image as in Appendix 2, then there is a simple relation between the 
epipolar lines on the two screens. A point on the left screen will lie on a unique epipolar 
line which will therefore correspond to a unique epipolar line in the right screen. 

We consider a general point in space (X, Y, Z). On the right screen, this projects to (Xi,Zi) 
by equation 3.13, 

Tfi 

X, = T ~-7 ^ -r -^—-(-/sin^+Xsin^-ycos^) (6.1) 

/ cos <pi — X cos fa — Y sin <px 

7n = -. -7 v mZ , v ■ , (6.2) 

/ cos <pi — X cos <pi — Y sin <j>x 



^»>-s, and on the left screen to (X 2 , Z 2 ) by (3.14), 



(6.3) 
(6.4) 



X 2 — — ■■■■■■■ - . -7 (/ sin fa +X sin fa - Y coa fa) 

f cos <p 2 + X cos <p 2 + Y sin fa 

7 ~mZ 

/ cos fa + X cos fa + Y sin fa 

To derive our relation, we use the four equations (6.1)-(6.4) to eliminate the unknowns X, 
Y and Z, which leaves us with a single equation relating Xx, X 2 , Z\ and Z%. 

We eliminate Z by dividing (6.2) by (6.4) to obtain 

Zi _ —{/ co s fa + Xcos^a + Ysxnfa) , R r » 

Z<i f cos<f>i — X cosfa — Y s\nfa ' 

We rewrite (6.1) and (6.3) respectively as 

X{ — X\ cos<^i — msin<£i} 

+ Y{— Xisin <f>i + mcoafa) 
+ Xx f cos 4> i + mf sin fa =0 

and 

X{ — X 2 cos fa — m sin fa} 

+ Y{— X2S111 fa + m cos fa} 
f^> — X>if cos fa — rnf sin fa — 

We combine (6.6) and (6.7) into a matrix equation 



(6.6) 



(6.7) 
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/— Xi cos^j — msin^i — Xi sin fa + in cos fa\fX\ f—X\f cos fa — mf sin fa\ . . 

^— X2 cos <^2 ~ "J sin <^2 —X 2 smfa + mcosfaJ\Y) \ X 2 fcosfa +mf sin fa) 

We invert the matrix to solve for X and Y obtaining* ' 

= f{(XjX 2 - m 2 )s in(< ^ + fa) - m{X v + X 2 )co S (fa + fa)} 

{X X X 2 + m 2 } sii.(^ 2 - fa) + m(X 2 - X t ) cos(<^ - <Ai) l ' 

and 

V— {X1X2 co s <ft 2 cos <ft t + mX 2 cos fa sin ^ t + mX t sin <ft 2 cos <ftt + ra 2 sin <ft 2 sin fa} / r ln \ 

_ .~ ' {X t X 2 + m 2 } sin(0 2 -(j>i) + m{X 2 -X 1 )cos{<l> 2 -<l>i) ( > 



We rewrite (6.5) as 



(Zi coa(f>i + Z 2 cos <f> 2 )f + [Z 2 cos <^2 — ^i cos fa)X 

+ (Z 2 sin fa — ^1 sin <£i)y = 0. 



(6.11) 



and substitute (6.9) and (6.10) into (6.11). After some manipulation, this simplifies and we 
obtain the result 

Zy {m sin fa + X 2 cos fa } = Z 2 {m sin fa + Xj cos fa } (6.12) 

Thus we see that if w 3 know the position on one screen, for example {Xi,Zi), wo find 
that the other position on the other screen, (X 2 ,Z 2 ), must lie on a straight line. ThisMs, of 
course, the standard epipolar line constraint. It is interesting to see that in our coordinate 
system, it can be written in such a simple form. This makes it straightforward to implement 
it in an algorithm. 



7. Outline of a contour based algorithm 

A simple example of an algorithm which indicates how the generalized ordering constraint 
can be exploited is as follows. 

Instead of using the epipolar lines to impose an ordering constraint, we use them as a 
consistency constraint. In conjunction with the ordering constraint along the contour, they 
determine stereo matching. This avoids wasteful scanning of all epipolar lines and ensures 
that computation is done only at places where it is necessary. It also automatically enforces 
figural continuity. There are clearly interesting effects when the contour we are matching 
runs along an epipolar line for some distance. Psychophysical experiments (Buelthoff and 
Poggio, pers. comm.) suggest that the human visual system also has problems in such 
cases. 

For a single contour the generalized ordering constraint tells us that the leftmost point 
and the rightmost point in the two images must correspond (see figure 9). These points can 
be used to "register" the viewing system, using the formulae in Appendix 2, and hence to 
determine implicitely the epipolar lines. Starting at one of these two points we move along 
the contours using the epipolar line constraint (6.12) to determine which points are to be 
matched. If more than one contour is present the generalized ordering constraint is used 
to decide which contours in the two images correspond (contours are also "ordered" in a 
similar way as points along a single contour are). In this case there will be many points in 
jf\ the two images which are known to correspond (two for each contour) and the registration 

process will hence be more robust (Mayhew and Longuet-Higgins, 1982). An algorithm of 
this type is currently under development with E. Tiffany. 
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f*\ 8. Discussion 

A few remarks about photometric effects and perspective invariants are relevant at 
this stage. We have assumed that the contours in the image being matched correspond 
to a physical structure, such as a wire or a silhouette, in the object being viewed. If 
zero-crossings of the Laplacian of a Gaussian are used as the matching primitives we must 
be sure they correspond to a precise physical location in the object being viewed. It is 
encouraging that results by Yuille (1984) suggest that many zero-crossings may be due 
to significant changes in the geometry of the object which are almost independent of the 
viewing positions. As it stands now, the theory presented here is only valid for wire-frame 
objects. Occluding contours, though theoretically "wrong", may however be often used in 
small angle stereo (Grimson, 1981). The main challenge is to extend the matching scheme 
presented here to solid and textured surfaces. Several possibilities can be considered: a) 
fingerprints representations (Yuille and Poggio, 1983) may provide specific features to be 
used in the matching process; b) several functional measurements of the two images (Krass, 
1984) along the contour may be used to perform the matching. 

The generalized ordering constraint analysis is done under the assumption of a three- 
dimensional curve in space. The analysis may be applied to occluding contours, but we 
need estimates of the errors. We also need to identify which contours are occluding 
contours. One possibility is to use fingerprints (Yuille and Poggio, 1983). The idea would 
then be to identify separately zero crossings corresponding to occluding contours, zero 
crossings corresponding to step edges and zero crossings due to texture. Some of these 
zero-crossings could then be used for a matching scheme based on the generalized ordering 
constraint. Interpolation of the surfaop hptweeo matched 7 ero crossino could be performed 
under the constraint of matching measurements provided by a scheme similar to Kass' 
(1984). His scheme may be reformulated to exploit a form of ordering constraint for speeding 
up the search. The main problem for a practical algorithm is obviously the stability of the 
contours to be matched between the two images. Geometric and photometric distortions 
are likely to present a hard problem. 

It is also possible to use perspective invariants to help match two contours which 
arise from the same physical location. Points on the two contours which correspond to the 
same invariant can be matched. A number of perspective invariants or "semi-invariants" 
are discussed by Yuille and Verri (1984). For example, the zeros and the discontinuities of 
curvature of a non-planar curve are preserved under perspective projection to the image 
plane, although the converse is not true. These "semi-invariants" can be used to find points 
on two contours which correspond. 

In summary, we have shown that, with the Longuet-Higgins (1982) geometry, a 
generalized ordering constraint holds provided the viewed object does not enter the 
forbidden zone. This constraint is not restricted to epipolar lines. We characterize the 
forbidden zone for orthographic and perspective projections and discuss experiments which 
suggest that the human visual system incorrectly interprets objects in this zone. Results 
by Burt and Julesz (1980) suggest that limits to fusion occur before this zone is entered. 
We use the generalized ordering constraint to propose an algorithm for stereo matching 
along contours using the epipolar lines as a consistency constraint. We use the generalized 
ordering constraint to determine points in the images which correspond and hence can 
be used to find the viewing parameters. The generalized ordering constraint provides an 
efficient way to match points by scanning only along the available contours instead of along 
all epipolar lines. It also automatically imposes a figural continuity constraint (Mayhew, 
1983). 



f*\ 
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/"""N Appendix 1: The Orthographic Limit of Perspective Projection 

From equation (3.7), we have the perspective projection of a point X on a screen with 
normal a it focus fi and focal length m is given by x-ff »vhere: 

— in 

Xi = (fT-~xy^" <(fl " x) _ {{h ~ x) ' ai) - i} {AAA) 

The term inside the brackets corresponds to the orthographic projection of (f x - X) on 
screen 1. It is scaled by a term (fi - X)-a,. Provided we stay in a region of the image 
where this term only varies a little, orthographic projection will be a good approximation 
and the scaling will be constant. This will normally be the case when (f - X) and a x are 
almost parallel, since we have 

(f-X)-fl 1 = |f-X|cosr (A1.2) 

where r is the angle between (f- X) and a t . For r near zero, coar is approximated by 
1 - r 2 /2 and will be insensitive to small changes in r. 

How good the orthographic approximation is in general will depend on the gradient of 
the surface being projected not being too large. 



/"^ 



^ 



Appendix 2: Solving for the Viewing Angles 

The projection Xi of a point X on screen 1 is given by equation (3.7), and we write it 



as 



YYi 

Xi = ma, - -— - t (X - f t ) (A2.1) 

A — Ti • a 

Alternately, we can take a point 1\ on screen 1 and construct the line joining it to the 
focus (Fx). The point P x has position vector Z x relative to the origin X of screen 1. From 
Figure (1) and Figure (8)we see that 

X = f ! - ma x [A.2.2) 

The gradient of the line from P x to F\ is mai - Zi, and we can write the line as 

r(X) = fi - maj + Zj + Xlma! - Z^ (A2.3) 

We rewrite (A.2.3) as 

r(X) = fi+(X-l){TO«i-Zi} (A2.4) 

and we obtain a similar equation for screen 2: 

r( M ) = f 3 + ( M - i){rna 2 - Z 2 } (A2.5) 
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(T*\ These lines intersect at a point P where 

f l + (X - l){ma t - Z t } = f 2 + (n - l){m« 2 - Z 2 } (A2.6) 

Now in our geometry there is a fixed (vertical) axis k such that 

2 1 .k = s 2 -k = fi-k=sf 2 -k = (A.2.7) 

I\ lies on screen 1 so Z ( • a x = and so we can define coordinates Zi(= Z x • k) and 
Xi of Z t by 

Zx = ^k + X x (k x aj (A2.8) 

Similarly, for Z 2 we have 

Z 2 = 2T 2 k + X 2 (k X « 2 ) (A2.9) 

Taking the dot product of (A. 2.6) with k yields 

= -(/i - \)Z 2 + (\ - l)Zi (A2.10) 

For convenience we replace X and n with f and f where 

f~\ f= -■ (A2.ll) 

f == /A — 1 

Then we rewrite (A. 2.6) using (A. 2.10) as 



f l — h = v(ma 2 — X 2 (k X a 2 )) 

-{(maj-J^kXSi)) 



and equation (A. 2. 10) as 



v Z x 



(A2.12) 



(A2.13) 



Substituting for f from (A.2.13) into (A. 2. 12) gives 

fi - h = v{{ma 2 - X 2 (k X a 2 )) - 4m 2l - Xi(k X ftj))} (A2.14) 

_Now we assume ajsecond point is also known to correspond. This has coefficients 
Z lt Xi on screen 1 and Z 2 ,X 2 on screen 2. Repeating the argument leading up to (A.2.14) 
gives the equation 

f i - f 2 = T{{ma 2 - X 2 (k X « 2 )) - ^(m ai - X t (k X &1 ))} (A2.15) 

for some constant r. 

Define 2f = f j — f 2 and the angles and <£ as in figure (14). Then taking the dot 
product of (A. 2.1 5) with «i and a 2 yields 
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<* s , 



o 



Figure 14 See tex/ 



and 



1 7 

-2/ cos(0 + 0) = m cos - X 2 smO - -£m (A.2.16) 

v Z% 



-2f ccr. i — m- ~ 2 -{rr. coc ■!- X A sin 0) (A.2.17) 



Similarly from (A. 2.1 5) we can obtain 



1 ~Z~ 

-2f cos(Q + <j>) = TOcos0-X 2 sin0- r^m (A.2.18) 



and 

2/ cos <£ = m ~(m cos {J + X~i sin (?) (A.2.19) 



1„. , ^2, 



Dividing (A.2.16) by (A.2.18) yields 



r m cos — X 2 sin — ^to 
" mcostf-X^sintf- f^TO 



Similarly, dividing (A.2.17) by (A. 2.19) yields 



r to — jr-(m cos + Xi sin0) 
" m- ^{m cos O + Xi sin 0) 



(A2.20) 



(A2.21) 



Hence we can combine (A. 2.20) with (A. 2.21) to yield an equation in only. 

Z 7j 

(to cos — Xn sin — —m)(m — — (to cos + Xi sin 0)) = 

^ £ 5 (/i - 2 - 22) 

(to cos — X2 sin — —m)(m — — -(to cos + Xi sin 0)) 
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O Set 



O 



/"""N 






zT' B= ^r ( A - 2 - 2 3) 



Then, after some manipulation we write (A. 2.22) as 

{m 2 (>l - 5) + {AX{X 2 - DXiX-i)} sin 2 

+ m{(X 2 - X 2 ) + ABpCi - Xi)} sin (yt.2.24) 

+ m{/?(X x - X 2 ) + A(X 2 - X x )} sin cos = 

We can divide this equation by sinO ( sin0 = corresponds to the special case when 
the eyes are fixating at infinity) and obtain an equation of form 

<7i sin 9 + C 2 + C 3 cos $ = (A.2.25) 

where 

Ci = m 2 (A - B) + (AXiX 2 - BX\X 2 ) 

<? 2 = m {X" 2 - X 2 + AB(X l - X t )} (A.2.26) 

G 3 = m^Xi - X 2 ) + A(X" 2 - Xi)} 

(A.2.25) can be written as a quadratic equation 

{C\ + C\) cos 2 + 2<7 2 <? 3 cos + {C\ - C\) = (A.2.27) 

and has solution 

COS = -g.gi±gi>/ci + c;-oi ,. 2 28 , 

c tf cf -f-ci (A2 - 28) 

_ JNote that (<7i,<7 2 ,C 3 ) ~ (— <7 lf — C a ,— C 3 ) as (A,B) ^ (B,A) and (X lf X 2 ,Yi > X'a) ^ 
(Xi,X 2 ,X],X 2 ). So if we have two_ points in the visual field which are known to coincide 
then we know that points P t _and P t on screenj correspond^ to P 2 and P 2 on screen 2. 
The coefficients (X ) ,Z 1 ),(3' 1 ,Zi), (X 2 ,i? 2 ) and {X 2 ,Z 2 ) of Pi,P u P 2 and P 2 are known and 
hence from (A.2.23) and (A.2.26), Cj,C 2 and C 3 are known. (A.2.28) gives two possible 
solutions for in terms of C X ,C 2 and C 3 . Let these be 0i and 2 . Dividing (A.2.16) by 
(A.2.1 7) yields 

cos(fl + <j>) _ rn cos 6 - X 2 sin - Z 2 /Zim 

cos<£ m — Z 2 /Zi(mcoa0 + Xisia0) ( • • ) 

Expanding cos(0 + <j>) = cos Ocos<j>- sin sin <£ we can rewrite (A. 2.29) as 

~C X sin <j> + V 3 cos <£ = (A.2.30) 



where 



(?i — in — Z 2 jZ\ (m cos + Xi sin 0) 
C 3 — — X 2 — Z 2 /Zi(rn sin — Xi cos 0) 



(A.2.31) 
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jT"*S There are two possible values of (^1,^2,^3) depending on whether — t or = 2 . 

(A. 2.31) will give two possible solutions for <f>, for each value of 0: 







±c t \/cl + cl 

«**=- J (A2.32) 

o t + o 3 

Hence we have four possible solutions for and <t> which can be formed directly from 
(A. 2.28) and (A. 2.29). In the general case, only one of these solutions will be physically 
1 reasonable and the others can be discarded. 

'iJ^ 1 " We have shown that if the images on both screens of points in space are known then 

" ' we can solve the non-linear equations directly for the angles and <j> that specify the stereo 
y geometry. These solutions are given by (A. 2.28) and (A.2.32). Two solutions are generated 
of which only one is physically reasonable in general. 
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